MASKED LANGUAGE MODELS

Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)

Challenge:	Language models (LMs) often include societal biases encoded in the human-produced datasets used for their training.
Approach:	They evaluated six prominent language models: BERT, RoBERTa, DistilBERT, BERT- multilingual, XLM-RoBERT and DistilberT- multilinguistic.
Outcome:	The results show that the models generated by the models were stereotypically gendered and with a reduced bias in multilingual variants.

Copied to clipboard

Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, Trevor Cohn

A Thorough Evaluation of Task-Specific Pretraining for Summarization (2021.emnlp-main)

Challenge:	Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively.
Approach:	They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models.
Outcome:	The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation.

Copied to clipboard

Sascha Rothe, Joshua Maynez, Shashi Narayan

Information Extraction & Text Mining

Challenge:	Previous work has used task-agnostic pretraining methods like masked language models or corrupted span prediction to improve performance on downstream tasks.
Approach:	They propose to use a task-agnostic pretraining to improve on low-resource tasks.
Outcome:	The proposed model can predict extracted gap sentences on summarization with a low resource and zero shot setup.

Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies (2023.acl-long)

Copied to clipboard

Iria de-Dios-Flores, Juan Garcia Amboage, Marcos Garcia

Open-ended Long Text Generation via Masked Language Modeling (2023.acl-long)

Challenge:	Using psycholinguistic and computational experiments, we compare the ability of humans and several pre-trained masked language models to correctly identify control dependencies in Spanish sentences.
Approach:	They compare the ability of humans and several pre-trained masked language models to correctly identify control dependencies in Spanish sentences such as ‘José le prometió/ordenó a Mara ser ordenado/a’.
Outcome:	The models fail to identify the correct antecedent in non-adjacent dependencies, showing their reliance on linearity.

Copied to clipboard

Xiaobo Liang, Zecheng Tang, Juntao Li, Min Zhang

Cross-lingual Similarity of Multilingual Representations Revisited (2022.aacl-main)

Challenge:	Pre-trained autoregressive language models have dominated OPen-ended Long Text Generation (Open-LTG) however, the low inference efficiency of AR impedes their usability.
Approach:	They propose a representative iterative non-autoregressive (NAR) decoding strategy to improve inference efficiency for Open-LTG.
Outcome:	The proposed model can generate short text and collapse for long text modeling.

Copied to clipboard

Maksym Del, Mark Fishel

Text Adversarial Purification as Defense against Adversarial Attacks (2023.acl-long)

Challenge:	Similarity indexes like CKA and CCA are not suitable for cross-lingual learning analysis.
Approach:	They propose an alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context.
Outcome:	The proposed method is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context.

Copied to clipboard

Linyang Li, Demin Song, Xipeng Qiu

Robustness in NLP

Challenge:	Existing methods for defending against adversarial examples are difficult due to the discrete nature of texts.
Approach:	They propose a novel adversarial purification method that aims to remove adversarials and make correct predictions based on the recovered clean samples.
Outcome:	The proposed method can defend against word-substitution adversarial attacks using language models.

Pre-Training Transformers as Energy-Based Cloze Models (2020.emnlp-main)

Copied to clipboard

Kevin Clark, Minh-Thang Luong, Quoc Le, Christopher D. Manning

Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification (2022.starsem-1)

Challenge:	elucidates close connection between cloze modeling and representation learning over text.
Approach:	They propose an energy-based cloze model for representation learning over text . they assign a scalar energy score to each input token indicating how likely it is given context .
Outcome:	The proposed model performs better than masked language models and faster than cloze models.

Copied to clipboard

Ryosuke Takahashi, Ryohei Sasano, Koichi Takeda

On Evaluating and Mitigating Gender Biases in Multilingual Settings (2023.findings-acl)

Challenge:	Recent research shows that contextualized word embeddings can give promising results for idiom token classification.
Approach:	They propose to leverage contextualized word embeddings from masked language models to improve idiom token classification.
Outcome:	The proposed method improves idiom token classification for English and Japanese datasets.

Copied to clipboard

Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram

A Simple Yet Effective Hybrid Pre-trained Language Model for Unsupervised Sentence Acceptability Prediction (2022.aacl-short)

Challenge:	Existing benchmarks and resources for evaluating gender biases in multilingual settings are limited.
Approach:	They propose to extend DisCo to different Indian languages using human annotations to evaluate gender biases in multilingual models.
Outcome:	The proposed benchmarks and mitigation techniques are extended beyond English to evaluate gender biases in multilingual models.

Copied to clipboard

Yang Zhao, Issei Yoshida

Garden Path Recovery in Causal and Masked Language Models (2026.acl-srw)

Challenge:	Existing unsupervised prediction approaches rely on language models to estimate sentence acceptability . low-frequency words would have a significant negative impact on sentence likelihood .
Approach:	They propose a method that substitutes Part-of-Speech (POS) tags for low-frequency words in sentences . their method improves both a sentence acceptability benchmark and a cross-domain sentence evaluation corpus .
Outcome:	The proposed method improves on a sentence acceptability benchmark and a cross-domain sentence evaluation corpus.

Copied to clipboard

Sanjan Baitalik, Rajashik Datta

Transparency Helps Reveal When Language Models Learn Meaning (2023.tacl-1)

Challenge:	a linguistics study of garden-path sentences shows that recovery dynamics are important for linguistic evaluation . causal models show larger within-model disambiguation effects than masked models overall .
Approach:	They propose to compare garden-path recovery in causal and masked language models . they use 100 English garden- path/control pairs spanning three constructions .
Outcome:	The proposed model shows that decoder-only models exhibit sharper disruption at the point of syntactic revision, while encoders appear comparatively buffered at the disambiguator due to right-context access.

Copied to clipboard

Zhaofeng Wu, William Merrill, Hao Peng, Iz Beltagy, Noah A. Smith

Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)

Challenge:	Existing language models are trained to optimize unsupervised objectives on text . despite their centrality, current models do not represent natural language semantics well .
Approach:	They show that autoregressive and masked language models learn to emulate semantic relations between expressions when context-dependent . they argue that a learner that has access to all Java code can never learn execution .
Outcome:	a new study shows that language models fail to represent natural language semantics well . the authors show that the model learning fails when denotations are changed to be context-dependent .

Copied to clipboard

Seiji Sugiyama, Risa Kondo, Tomoyuki Kajiwara, Takashi Ninomiya

Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)

Challenge:	Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages.
Approach:	They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences .
Outcome:	The proposed method can be used on four sentence pair modeling tasks in English and Japanese.

Copied to clipboard

Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew Leavitt

Debiasing by obfuscating with 007-classifiers promotes fairness in multi-community settings (2025.coling-main)

Challenge:	Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate.
Approach:	They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively.
Outcome:	The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines.

Copied to clipboard

Ingroj Shrestha, Padmini Srinivasan

Mixed-Lingual Pre-training for Cross-lingual Summarization (2020.aacl-main)

Challenge:	a number of studies have focused on the mitigation of biases in text classifiers.
Approach:	They propose an obfuscation-based data augmentation debiasing approach to reduce bias . they add to the training data obfuses versions of all false positive instances .
Outcome:	The proposed approach reduces bias for almost all of the tests without sacrificing false positive rates or F1 scores for minority or majority communities.

Copied to clipboard

Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, Xuedong Huang

Text Generation

Challenge:	Cross-lingual summarization (CLS) aims at producing a summary in the target language for an article in the source language.
Approach:	They propose a mixed-lingual pre-training scheme that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models.
Outcome:	The proposed model improves on the translation and masked language models with no task-specific components and saves memory.

Probing Physical Reasoning with Counter-Commonsense Context (2023.acl-short)

Copied to clipboard

Kazushi Kondo, Saku Sugawara, Akiko Aizawa

Reasoning

Challenge:	Recent language models have succeeded in tasks that do not require contextual reasoning, such as size comparison and prediction of event frequency.
Approach:	They propose to use a dataset to measure physical commonsense to infer size relationships between objects in contexts that fit physical common sense.
Outcome:	The proposed dataset consists of contexts that fit physical commonsense and those that don't . results show that language models fail to use verbs and make incorrect judgments .

Improving Contextual Representation with Gloss Regularized Pre-training (2022.findings-naacl)

Copied to clipboard

Yu Lin, Zhecheng An, Peihao Wu, Zejun Ma

LLM Bias Detection and Mitigation through the Lens of Desired Distributions (2025.emnlp-main)

Challenge:	Experimental results show that the gloss regularizer module enhances word semantic similarity in pre-training.
Approach:	They propose an auxiliary gloss regularizer module to BERT pre-training to enhance word semantic similarity.
Outcome:	The proposed model improves word similarity in word-level and sentence-level representation.

Copied to clipboard

Ingroj Shrestha, Padmini Srinivasan

A Better Way to Do Masked Language Model Scoring (2023.acl-short)

Challenge:	Prior work on bias mitigation has focused on promoting social equality and demographic parity, but less attention has been given to aligning LLM’s outputs to desired distributions.
Approach:	They propose a weighted adaptive loss based fine-tuning method that aligns LLM’s gender–profession output distribution with the desired distribution while preserving language modeling capability.
Outcome:	The proposed method achieves near-complete mitigation under equality and 30–75% reduction under real-world settings.

Copied to clipboard

Carina Kauf, Anna Ivanova

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020.emnlp-main)

Challenge:	Existing methods to estimate log-likelihood of sentences are not available for masked language models . et al., 2020: e.on.org/sci/sql/src/spll-word-l2r estimates sentence pseudo-log-likellihoods for a given sentence under autoregressive models are difficult to perform on controlled benchmarks.
Approach:	They propose to estimate sentence pseudo-log-likelihood scores by successively masking each sentence token, retrieving its score using the rest of the sentence as context, and summing the resulting values.
Outcome:	The proposed method outperforms both the original and the adapted method on masked language models.

Copied to clipboard

Nathan Ng, Kyunghyun Cho, Marzyeh Ghassemi

Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution (2020.coling-main)

Challenge:	Data augmentation is a common method used to improve out-of-domain (OOD) generalization.
Approach:	They propose a data augmentation method that uses corruption and reconstruction functions to move randomly on a manifold to generate training examples.
Outcome:	The proposed method outperforms existing methods and baseline models on both in-domain and OOD data and achieves gains of 0.8% on OOD Amazon reviews, 1.8% accuracy on OOO MNLI, and 1.4 BLEU on in- domain IWSLT14 German-English.

Copied to clipboard

Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, Alexander Panchenko

BabyLM’s First Constructions: Causal interventions provide a signal of learning (2025.emnlp-main)

Challenge:	Lexical substitution is a powerful technology used in various NLP applications . it generates plausible words that can replace a given word in a textual context .
Approach:	They propose to use a large-scale comparative study to compare lexical substitution methods . they compare existing and new methods using word sense induction datasets .
Outcome:	The proposed methods improve competitive results by incorporating information about the target word into the models.

Copied to clipboard

Joshua Rozner, Leonie Weissweiler, Cory Shain

Word Order Matters When You Increase Masking (2022.emnlp-main)

Challenge:	Recent work shows sensitivity to constructions in pretrained language models, but their relevance to human language learning is doubted.
Approach:	They use construction grammars to demonstrate sensitivity to constructions in pretrained language models.
Outcome:	The proposed models learn diverse constructions even hard cases that are superficially indistinguishable.

Copied to clipboard

Karim Lasri, Alessandro Lenci, Thierry Poibeau

Gendered Mental Health Stigma in Masked Language Models (2022.emnlp-main)

Challenge:	Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks.
Approach:	They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences.
Outcome:	The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks.

Copied to clipboard

Inna Lin, Lucille Njoo, Anjalie Field, Ashish Sharma, Katharina Reinecke, Tim Althoff, Yulia Tsvetkov

Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (2021.acl-long)

Challenge:	Mental health stigma prevents many individuals from receiving appropriate care, and social psychology studies have shown that mental health tends to be overlooked in men.
Approach:	They propose to use clinical psychology literature to curate prompts, then evaluate models’ propensity to generate gendered words.
Outcome:	The proposed framework captures stigma about gender in mental health and is more likely to predict female subjects than male in sentences about mental health conditions (32% vs. 19%), and this disparity is exacerbated for sentences that indicate treatment-seeking behavior.

Copied to clipboard

Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, Jin Xu

Trained on 100 million words and still in shape: BERT meets British National Corpus (2023.findings-eacl)

Challenge:	Recent studies show that pre-trained masked language models can be factual knowledge bases.
Approach:	They conduct a rigorous study to explore the underlying predicting mechanisms of MLMs . they find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts a .
Outcome:	The proposed model improves on illustrative cases and external contexts . the results question the previous findings that MLMs can be reliable factual knowledge bases .

Copied to clipboard

David Samuel, Andrey Kutuzov, Lilja Øvrelid, Erik Velldal

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020.emnlp-main)

Challenge:	masked language models are trained on ever larger corpora, but pre-training on a modestly-sized but representative, well-balanced, and publicly available corpus can reach better performance than the original BERT model.
Approach:	They propose an optimized LM architecture called LTG-BERT that can be used to train a competitive language model on a small and standardizable corpus.
Outcome:	The proposed architecture outperforms the original English BERT model on a representative, well-balanced and publicly available corpus.

Copied to clipboard

Nikita Nangia, Clara Vania, Rasika Bhalerao, Samuel R. Bowman

Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021.eacl-main)

Challenge:	Pretrained language models use cultural biases implicitly, causing harm . identifying and quantifying learnt biase enables us to measure progress .
Approach:	They propose a benchmark to measure social bias in pretrained language models . they use 1508 examples that cover stereotypes dealing with nine types of bias .
Outcome:	The proposed benchmark focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups.

Copied to clipboard

Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, Yulan He

Simple and Effective Multi-Token Completion from Masked Language Models (2023.findings-eacl)

Challenge:	Biomedical question-answering (QA) provides users with high-quality information from a vast scientific literature.
Approach:	They propose to use a biomedical entity-aware masking strategy to fine-tune masked language models to their domains.
Outcome:	The proposed approach is an adaptation process for masked LMs, not memory or components.

Copied to clipboard

Oren Kalinsky, Guy Kushilevitz, Alexander Libov, Yoav Goldberg

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions (2021.findings-acl)

Challenge:	Pre-trained neural masked language models are limited to predicting a single token . recent pre-tried LMs like T5 do allow predicting multi-token completions, but are more expensive to train and run.
Approach:	They propose two ways to adapt pre-trained masked language models to produce multi-token completions.
Outcome:	The proposed method surpasses current state-of-the-art models while being more parameter efficient.

Copied to clipboard

Nam Do, Ellie Pavlick

Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)

Challenge:	Existing studies have shown that pre-trained language models encode commonsense relational knowledge, but they are often insensitive to context, ignoring overt contextual cues such as negations.
Approach:	They propose a procedure that exploits generic associations in masked language models to create model-specific entailment schemas.
Outcome:	The proposed procedure exploits generic associations in masked language models to create model-specific Winograd-style entailment schemas.

Copied to clipboard

Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, Lei Li

Low-Cost Generation and Evaluation of Dictionary Example Sentences (2024.naacl-long)

Challenge:	masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations.
Approach:	They propose a representation learning approach that uses embeddings as anchors to model contextual representations.
Outcome:	The proposed model achieves 5x speedup and 1.2 points average improvement over MLM.

Copied to clipboard

Bill Cai, Ng Clarence, Daniel Liang, Shelvia Hotama

Do dialogue representations align with perception? An empirical study (2023.eacl-main)

Challenge:	Prior studies have shown that language models can be trained to generate example sentences, but they relied on costly customized models and word sense datasets for generation and evaluation.
Approach:	They propose a new automatic evaluation metric called OxfordEval that measures the win-rate of generated sentences against existing Oxford Dictionary sentences.
Outcome:	The proposed model achieves over 85.1% win rate against baseline sentences compared to 39.8% win rate for prior model-generated sentences.

Copied to clipboard

Sarenne Wallbridge, Peter Bell, Catherine Lai

Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering (2022.findings-emnlp)

Challenge:	masked language models produce stronger correlations than auto-regressive models, but humans and models make different response selection mistakes.
Approach:	They propose to use spoken conversation as a model to measure human comprehension behaviour.
Outcome:	The proposed model outperforms the model which produces the strongest correlation with human responses.

Copied to clipboard

Xuan Luo, Chuang Fan, Yice Zhang, Wanguo Jiang, Bing Qin, Ruifeng Xu

Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2022.acl-long)

Challenge:	Empirical results show that pre-trained language models can improve the typical answer generation of GLMs.
Approach:	They propose a ranking strategy that exploits WordNet to train a ranker that picks out the most popular answers for commonsense questions.
Outcome:	The proposed ranking strategy is tested on a commonsense question answering (QA) dataset and on negative samples from WordNet.

Copied to clipboard

Meng Cao, Yue Dong, Jackie Cheung

Text Generation

Challenge:	State-of-the-art abstractive summarization systems often generate hallucinations, i.e., content that is not directly inferable from the source document.
Approach:	They propose a detection approach that separates factual from non-factual hallucinations of entities by masked language models.
Outcome:	The proposed method outperforms baselines in accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks.

Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)

Copied to clipboard

Jiaxi Li, Wei Lu

Pre-trained Token-replaced Detection Model as Few-shot Learner (2022.coling-1)

Challenge:	a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers .
Approach:	They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span .
Outcome:	The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings.

Copied to clipboard

Zicheng Li, Shoushan Li, Guodong Zhou

Probing BERT’s priors with serial reproduction chains (2022.findings-acl)

Challenge:	Pre-trained masked language models have demonstrated remarkable few-shot learning ability . a novel approach to few- shot learning with pre-tried token-replaced detection models is proposed .
Approach:	They propose a method to reformulate a classification or regression task as a token-replaced detection problem by using pre-trained token-based models.
Outcome:	The proposed approach outperforms pre-trained masked language models in learning tasks . it can learn models with a few examples and generalize well from limited examples like humans .

Copied to clipboard

Takateru Yamakoshi, Thomas Griffiths, Robert Hawkins

Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation (2022.naacl-main)

Challenge:	Large neural language models have induced surprisingly human-like linguistic knowledge, from syntactic structure and subtle lexical biases to more insidious social biase and stereotypes.
Approach:	They propose to use serial reproduction chains to generate representative samples from popular masked language models like BERT to test their hypothesis.
Outcome:	The proposed method is based on theories of iterated learning in cognitive science and can be used to probe masked language models.

Copied to clipboard

Siyu Lai, Zhen Yang, Fandong Meng, Xue Zhang, Yufeng Chen, Jinan Xu, Jie Zhou

Robustness in NLP

Challenge:	Existing approaches to generate adversarial examples for NMT use the meaning-preserving restriction.
Approach:	They propose a new definition for adversarial examples based on the Doubly Round-Trip Translation (DRTT) they introduce masked language models to construct bilingual adversarials based upon DRTT .
Outcome:	The proposed approach significantly improves the robustness of the NMT model on clean and noisy test sets.

Large Scale Substitution-based Word Sense Induction (2022.acl-long)

Copied to clipboard

Matan Eyal, Shoval Sadde, Hillel Taub-Tabib, Yoav Goldberg

Automatic Reconstruction of Ancient Chinese Pronunciations (2024.findings-emnlp)

Challenge:	Word forms are ambiguous, and derive meaning from the context in which they appear . word sense induction can be performed over a corpus-derived sense inventory .
Approach:	They propose a word-sense induction method based on pre-trained masked language models . they train a static word embeddings algorithm on the sense-tagged corpus .
Outcome:	The proposed method outperforms existing senseful embeddings methods on Wikipedia and on an outlier detection dataset.

Copied to clipboard

Zhige Huang, Haoan Jin, Mengyue Wu, Kenny Zhu

Syntactic Text Processing

Challenge:	A human language is comprised of a pronunciation system and a writing system, both evolving and changing over time.
Approach:	They reformulate existing phonetic rules into a dataset of 70,943 entries for 17,001 Chinese characters and use it to perform a temporal prediction task.
Outcome:	The transformer-based model significantly advances the digitization and computational reconstruction of ancient Chinese phonology, providing a more complete and temporally contextualized resource for computational linguistics and historical research.

Self-Alignment Pretraining for Biomedical Entity Representations (2021.naacl-main)

Copied to clipboard

Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, Nigel Collier

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (2020.emnlp-main)

Challenge:	Existing approaches to self-supervised learning of biomedical entities are limited in the biomedic domain.
Approach:	They propose a pretraining scheme that self-aligns the representation space of biomedical entities.
Outcome:	The proposed framework achieves state-of-the-art on six MEL benchmarking datasets.

Copied to clipboard

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh

InforMask: Unsupervised Informative Masking for Language Model Pretraining (2022.emnlp-main)

Challenge:	Pretrained language models have been successful when finetuned to downstream tasks . however, it is difficult to determine whether the knowledge that finetuning LMs contain is learned during the pretraining or the finetailing process.
Approach:	They propose a method to create prompts for a diverse set of tasks using a gradient-guided search.
Outcome:	The proposed method performs sentiment analysis and natural language inference without additional parameters and finetuning.

Copied to clipboard

Nafis Sadeq, Canwen Xu, Julian McAuley

PROST: Physical Reasoning about Objects through Space and Time (2021.findings-acl)

Challenge:	Masked language modeling is used for pretraining large language models for knowledge-intensive tasks.
Approach:	They propose an unsupervised masking strategy that exploits Pointwise Mutual Information to select the most informative tokens to mask.
Outcome:	The proposed strategy outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.

Copied to clipboard

Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, Katharina Kann

Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure (2024.naacl-long)

Challenge:	a new dataset is available to test pretraining of physical reasoning models . state-of-the-art models are inadequate at reasoning about physical interactions, authors say .
Approach:	They present a dataset that contains 18,736 multiple-choice questions from 14 templates . they propose to use the dataset to probe both causal and masked language models .
Outcome:	The proposed dataset contains 18,736 multiple-choice questions covering 10 physical reasoning concepts.

Copied to clipboard

David Arps, Laura Kallmeyer, Younes Samih, Hassan Sajjad

Syntactic Text Processing

Challenge:	a number of studies have focused on making explicit the linguistic information encoded in language models (LMs) however, this method has been criticized for various reasons.
Approach:	They introduce a framework for creating nonce treebanks for multilingual UD corpora . they investigate word co-occurrence statistics and show how nonce data affects the performance of syntactic dependency probes.
Outcome:	The proposed framework satisfies syntactic argument structure and ensures grammaticality via language-specific rules.

Evaluation of HTR models without Ground Truth Material (2022.lrec-1)

Copied to clipboard

Phillip Benjamin Ströbel, Martin Volk, Simon Clematide, Raphael Schwitter, Tobias Hodel, David Schoch

DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain (2024.lrec-main)

Challenge:	Optical Character Recognition (OCR) is a well-established technique for digitising historical printed collections in libraries and archives.
Approach:	They propose to use masked language models to evaluate handwritten text recognition models . they propose to introduce GT-free metrics to evaluate models to ensure best results .
Outcome:	The proposed model evaluations are based on lexicon-based and masked language models.

Copied to clipboard

Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacome Constant Dit Beaufils, Natalia Grabar, Béatrice Daille, Solen Quiniou, Emmanuel Morin, Pierre-Antoine Gourraud, Richard Dufour

Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models (2020.emnlp-main)

Challenge:	Existing benchmarks for pre-trained language models are limited to only a few languages . a limited number of tasks are evaluated on non-standardized protocols .
Approach:	They propose to aggregate diverse downstream tasks into a benchmark to assess PLMs' qualities . they evaluate 8 pre-trained masked language models on general and biomedical-specific data .
Outcome:	The proposed benchmark assesses pre-trained language models on 20 diversified tasks.

Copied to clipboard

Thuy-Trang Vu, Dinh Phung, Gholamreza Haffari

BERT-ATTACK: Adversarial Attack Against BERT Using BERT (2020.emnlp-main)

Challenge:	Recent work has shown the importance of training contextualised word embedding models on the domain of the target task of interest.
Approach:	They propose a masking strategy which adversarially masks out those tokens which are harder to reconstruct by the underlying MLM.
Outcome:	The proposed training strategy outperforms random masking on six unsupervised domain adaptation tasks and achieves up to +1.64 F1 score improvements.

Copied to clipboard

Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, Xipeng Qiu

Robustness in NLP

Challenge:	Current approaches to generate adversarial samples for discrete data are heuristic replacement strategies that are difficult to implement in continuous data.
Approach:	They propose a method to generate adversarial samples using pre-trained masked language models using BERT.
Outcome:	The proposed method outperforms state-of-the-art methods in success rate and perturb percentage while remaining fluent and semantically preserved.

Coordination Generation via Synchronized Text-Infilling (2022.coling-1)

Copied to clipboard

Hiroki Teranishi, Yuji Matsumoto

Reconstruction Probing (2023.findings-acl)

Challenge:	Generating synthetic data from pre-trained language models has enhanced performance across several NLP tasks.
Approach:	They propose a method for generating sentences with a coordinate structure in which the boundaries of its conjuncts are explicitly specified.
Outcome:	The proposed method produces promising coordination instances that provide gains for the task in low-resource settings.

Copied to clipboard

Najoung Kim, Jatin Khilnani, Alex Warstadt, Abdelrahim Qaddoumi

Challenge:	a new analysis method for contextualized representations is proposed . contextualization boosts reconstructability of tokens close to the token being reconstructed .
Approach:	They propose a method for contextualized representations based on reconstruction probabilities in masked language models.
Outcome:	The proposed method compares reconstruction probabilities of tokens in masked language models . it finds that contextualization boosts reconstructability of token that are close to the token being reconstructed .

BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)

Copied to clipboard

Haoran Xu, Benjamin Van Durme, Kenton Murray

Solving Cosine Similarity Underestimation between High Frequency Words by ℓ2 Norm Discounting (2023.findings-acl)

Challenge:	Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance.
Approach:	They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings.
Outcome:	The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model.

Copied to clipboard

Saeth Wannasuphoprasit, Yi Zhou, Danushka Bollegala

DP-MLM: Differentially Private Text Rewriting Using Masked Language Models (2024.findings-acl)

Challenge:	Existing methods to estimate cosine similarity between words have been proposed, but no solution has been proposed.
Approach:	They propose a method to discount the l2 norm of contextualised word embedding by the frequency of that word in a corpus when measuring the cosine similarities between words.
Outcome:	The proposed method underestimates the similarity scores between two words when used with contextualised token embeddings from masked language models such as BERT.

Copied to clipboard

Stephen Meisenbacher, Maulik Chevli, Juraj Vladika, Florian Matthes

Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks (2022.emnlp-main)

Challenge:	Existing methods for text privatization using Differential Privacy rely on autoregressive models which lack a mechanism to contextualize the private rewriting process.
Approach:	They propose a method for differentially private text rewriting using masked language models to rewrite a text one token at a time.
Outcome:	The proposed method preserves utility at lower levels, compared to previous methods relying on autoregressive models with a decoder.

Copied to clipboard

Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, Reza Shokri

Learning to Substitute Words with Model-based Score Ranking (2025.naacl-long)

Challenge:	Prior attempts at measuring leakage of MLMs via membership inference attacks have been inconclusive, implying potential robustness of Mlms to privacy attacks.
Approach:	They propose a stronger membership inference attack based on likelihood ratio hypothesis testing that involves an additional MLM to more accurately quantify the privacy risks of memorization in MLMs.
Outcome:	The proposed attack improves the AUC of prior membership inference attacks from 0.66 to an alarmingly high 0.90 level on models trained on medical notes.

Copied to clipboard

Hongye Liu, Ricardo Henao

Revisiting Token Dropping Strategy in Efficient BERT Pretraining (2023.acl-long)

Challenge:	Experimental results show that the proposed approach outperforms both masked language models and large language models.
Approach:	They propose a model-based scoring approach to quantify sentence quality . they propose 'loss function' that optimizes alignment between model predictions and sentence scores .
Outcome:	The proposed approach outperforms masked language models and large language models in the quantitative analysis of word substitutions.

Copied to clipboard

Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, Dacheng Tao

Debiasing Large Language Models with Structured Knowledge (2024.findings-acl)

Challenge:	Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT.
Approach:	They propose a semantic-consistent learning method to improve token dropping by skipping the computation of a subset of input tokens at several middle layers.
Outcome:	The proposed method achieves consistent and significant performance gains across all tasks and model sizes.

Copied to clipboard

Congda Ma, Tianyu Zhao, Manabu Okumura

FrameEOL: Semantic Frame Induction using Causal Language Models (2025.findings-emnlp)

Challenge:	Existing methods to reduce biases in pre-training models are hampered by their performance.
Approach:	They propose a method that utilizes structured knowledge to mitigate bias in LLMs . their method obviates the need for training from scratch, thus offering enhanced scalability .
Outcome:	The proposed method outperforms state-of-the-art (SOTA) baselines in the debiasing ability.

Copied to clipboard

Chihiro Yano, Kosuke Yamada, Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda

Unsupervised Text Style Transfer with Padded Masked Language Models (2020.emnlp-main)

Challenge:	Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke.
Approach:	They propose a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label .
Outcome:	The proposed method outperforms existing methods on English and Japanese datasets.

Copied to clipboard

Eric Malmi, Aliaksei Severyn, Sascha Rothe

Low-Resource NLP

Challenge:	Existing methods for style transfer are difficult to obtain and require substantial amounts of parallel training examples to work well.
Approach:	They propose an unsupervised method for style transfer that uses masked language models to find the text spans where the two models disagree the most in terms of likelihood.
Outcome:	The proposed method performs competitively in a fully unsupervised setting and improves accuracy in low-resource settings by over 10 percentage points when pre-training on silver training data generated by Masker.

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020.emnlp-main)

Copied to clipboard

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi

Multimodality

Challenge:	Recent work has adapted vision-and-language models to generative tasks like image captioning.
Approach:	They propose an extension to LXMERT with training refinements to generate images from text.
Outcome:	The proposed model can generate images from pieces of text while still being comparable to existing models.

How Far Is Too Far? Studying the Effects of Domain Discrepancy on Masked Language Models (2024.lrec-main)

Copied to clipboard

Subhradeep Kayal, Alexander Rakhlin, Ali Dashti, Serguei Stepaniants

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models (2022.emnlp-main)

Challenge:	Pre-trained masked language models perform strongly on a wide variety of NLP tasks.
Approach:	They propose a mechanism to quantify the difference in domains between the pre-trained model and the task and partition it using a cloze task.
Outcome:	The proposed model performs better on openly available e-commerce datasets than the original model on scientific and biomedical datasets.

Copied to clipboard

Mengzhou Xia, Mikel Artetxe, Jingfei Du, Danqi Chen, Veselin Stoyanov

Do Transformers Parse while Predicting the Masked Word? (2023.emnlp-main)

Challenge:	Pre-trained masked language models perform few-shot learning, but discriminative models like ELECTRA do not fit into the paradigm.
Approach:	They propose to use ELECTRA to train pre-trained models to score originality of target options without introducing new parameters.
Outcome:	The proposed model outperforms masked language models in a wide range of tasks without adding new parameters.

Copied to clipboard

Haoyu Zhao, Abhishek Panigrahi, Rong Ge, Sanjeev Arora

Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts (2024.lrec-main)

Challenge:	Existing studies show that pre-trained language models encode linguistic structures like parse trees while being trained unsupervised.
Approach:	They propose to train pre-trained language models to encode linguistic structures like parse trees while unsupervised.
Outcome:	The proposed model performs optimally for masked language modeling loss on the English PCFG.

Copied to clipboard

Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, Javier Torroba Marchante, Shilin Xie, Sergio E. Zanotto, Aurélie Névéol